1. Field of the Invention
The present invention relates to an image processing device, an image processing method, a recording medium, and a program, and more particularly relates to an image processing device, an image processing method, a recording medium, and a program, capable of correcting distortion due to focal plane shutter effects in a taken image.
2. Description of the Related Art
The principle of readout from a CCD image sensor 11 will be described with reference to FIGS. 1 through 2C.
The CCD image sensor 11 includes photodiodes 21, vertical CCDs 22, a horizontal CCD 23, and an output amplifier 24.
As shown in FIG. 2A, at the CCD image sensor 11, charges stored in the photodiodes 21 are transferred to the vertical CCDs 22, at the same timing for all pixels. Subsequently, as shown in FIG. 2B, charges are transferred one at a time from all vertical CCDs 22 arrayed horizontally, to the horizontal CCD 23.
Next, as shown in FIG. 2C, these charges for all pixels are transferred in the horizontal direction at the horizontal CCD 23, subjected to signal amplification at the output amplifier 24, and externally output as analog signals.
Then, once again, charges are transferred one at a time vertically from the vertical CCDs 22 to the horizontal CCD 23, following which all charges are transferred in the horizontal direction at the horizontal CCD 23, subjected to signal amplification at the output amplifier 24, and externally output as analog signals. This is repeated until all charges transferred from the photodiodes 21 to the vertical CCDs 22 (i.e., charges for all pixels) have been externally output.
Thus, with a CCD image sensor 11, charges stored in all of the photodiodes 21 are transferred to the vertical CCDs 22 all at once, so the point-in-time at which exposure ends is the same for all pixels, and further, the point-in-time at which exposure is started, i.e., the point-in-time at which light is cast on all photodiodes 21 and photoelectric conversion is started again is the same for all pixels. This is what is known as a “global shutter” operation.
Next, the principle of readout from a CMOS image sensor 41 will be described with reference to FIGS. 3 through 4C.
With a CMOS image sensor 41, charges stored in the photodiodes 21 are output to vertical signal lines 51 in the from of signal currents, via unshown in-pixel amp transistors and transfer transistors. The signal currents are then supplied to a column signal processing units 52, subjected to predetermined signal processing, and then externally output.
The photodiodes 21 corresponding to each pixel vertically arrayed is connected in common to the vertical signal line 51 thereof, so in order to read out the signal from each pixel independently, an arrangement has to be made wherein only one pixel worth of signal is output to the vertical signal line 51 each time.
That is to say, with a CMOS image sensor 41, signals are first read out from each of the photodiodes 21 corresponding to the pixels arrayed at the bottom row (head row) for example, as shown in FIG. 4A, following which the row being read is sequentially changed as shown in FIG. 4B, and finally signals are read out from each of the photodiodes 21 corresponding to the pixels arrayed at the top row (final row), as shown in FIG. 4C, thereby enabling signals for all of the pixels to be read out independently.
At this time, each photodiode 21 starts exposure again immediately following readout of the stored charge, so there is difference in the exposure start point-in-time and the exposure end point-in-time between the photodiode 21 of the head row and the photodiode 21 of the final row. This is a shutter action known as a focal-plane shutter action (or rolling shutter action).
The exposure start timing and the exposure end timing of the photodiodes 21 of each row, and the charge readout start timing, will be described with reference to FIGS. 5 and 6.
FIG. 5 illustrates a case wherein an electronic shutter is not employed, and FIG. 6 illustrates a case wherein an electronic shutter is employed. FIGS. 5 and 6 illustrate the relation between time axis, represented by the horizontal axis, and rows, represented by the vertical axis.
That is to say, since the readout timing of charges from the photodiodes 21 of each row cannot occur concurrently, readout of all pixels of the preceding frame, i.e., readout of the charges of the last row of the preceding frame, must be completed before the exposure time T1 elapses from starting of exposure of the photodiodes 21 of the leading row and reading of charges from the photodiodes 21 of the leading row starts.
For example, as shown in FIG. 5, in the event that no electronic shutter is used, in a case wherein the ending of readout of charge of the N'th frame stored in the photodiodes 21 of the last row (starting exposure of the N+1'th frame) and starting readout of charges of the N+1'th frame stored in the photodiodes 21 of the leading row (ending exposure of the N+1'th frame) occur at consecutive timings, the difference in the timing of starting exposure between the leading row and the last row is minimal, and is approximately equal to the exposure timing T1.
As shown in FIG. 6, in the case of using an electronic shutter, there is an invalid exposure period due to the electronic shutter, so the difference in exposure starting timing between the leading row and the final row is approximately equal to the sum of the exposure period T2 and the invalid exposure period due to the electronic shutter.
In the event of shooting a moving subject, or in the event of the camera itself changing angle (panning) during exposure, image distortion occurs due to exposure time offset due to focal plane shutter operations (or rolling shutter operations). The phenomenon wherein such distortion occurs is known as “focal plane shutter effects” (or “rolling shutter effects”).
A specific example of trouble occurring in a taken image due to focal plane shutter effects will be described with reference to FIGS. 7A through 8B.
Let us say that we are shooting an object rotating at high speed with four blades, as a subject. FIG. 7A illustrates an image obtained by shooting with a global shutter operation wherein the exposure timings of all pixels match, such as with a CCD imaging device, and FIG. 7B illustrates an image obtained by shooting with focal plane shutter readout.
We will further say that with the focal plane shutter, the exposure start point-in-time shifts from the top of the taken image toward the bottom thereof, and that the subject is rotating in the counter-clockwise direction.
With the global shutter operations, the exposure timing for capturing the motion of the subject is identical for all pixels, so there may be cases wherein a moving subject appears blurred in the taken image depending on the shutter speed, but the shape of the subject itself can be captured in a true manner.
On the other hand, with the focal plane shutter operation, the exposure start point-in-time shifts in the direction of rotation for portions which move from the right to the left within the screen, so the farther down in the image, the greater the distortion of shape is toward the left. Also, portions which move from the top toward the bottom within the screen match the shift in exposure start point-in-time, so the farther down in the image, the wider the shape appears. For portions which move from the left to the right within the screen, the farther down in the image, the greater the distortion of shape is in the direction of motion, i.e., toward the right. Moreover, portions which move from the bottom toward the tow within the screen move opposite to the shift in exposure start point-in-time, so the farther up in the image, the narrower the shape appears.
Thus, in the event of taking a moving subject with an imaging device performing focal plane shutter operations, the obtained taken image is distorted according to the direction of movement.
Next, in a case of an example wherein the camera is moved (panned horizontally) at the time of taking a still subject, FIG. 8A illustrates an image obtained by shooting with a global shutter operation wherein the exposure timings of all pixels match, and FIG. 8B illustrates an image obtained by shooting with focal plane shutter readout. The situation illustrated here is a case wherein the camera is panned from the left to the right while shooting.
In this case as well, with the global shutter operations, the exposure timing for capturing the motion of the subject is identical for all pixels, so there may be cases wherein a moving subject appears blurred in the taken image depending on the shutter speed, but the shape of the subject itself can be captured in a true manner.
However, with focal plan shutter operations, the shape of the subject appears distorted in the direction of movement of the camera, due to the relation between movement of the camera and the offset in exposure start point-in-time.
Note that with the distortion of the subject due to the focal plane shutter operations described with FIGS. 7A through 8B, the greater the motion of the subject is, and/or the greater the motion of the camera is, the greater the distortion is. Even with a fast shutter speed, distortion will occur in the taken image of the motion of the subject or camera is great.
Accordingly, there has been developed a technique for alleviating image distortion due to focal plane shutter readout with a CMOS image sensor, for example by writing data from the CMOS image sensor to external memory faster than the frame rate and reading the written data out more slowly in accordance with the frame rate, thereby obtaining a taken image equivalent to that of global shutter operations (e.g., see Japanese Unexamined Patent Application Publication No. 2004-140479).
There have also been proposed a technique wherein transistors necessary for global shutter operations are added to the pixels (e.g., see Japanese Unexamined Patent Application Publication No. 2004-140149), a technique wherein capacitors necessary for global shutter operations are (e.g., see Japanese Unexamined Patent Application Publication No. 2004-159555), a technique wherein both transistors and capacitors necessary for global shutter operations are added (e.g., see Japanese Unexamined Patent Application Publication No. 2005-65074), and so forth.
Further, there has been proposed an image processing technique wherein, for example, image distortion obtained by shooting with focal plane shutter operations is corrected by comparison with a registered reference image (e.g., see Japanese Unexamined Patent Application Publication No. 58945).